Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save omitted rego evaulation results #1604

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

adhilto
Copy link
Collaborator

@adhilto adhilto commented Feb 28, 2025

🗣 Description

Save what the result would have been for controls that are omitted if they had not been omitted to the JSON output (HTML output is unchanged). Two new keys are added for each test result:

  • OmittedEvaluationResult: What the result would have been, e.g., "Fail"
  • OmittedEvaluationDetails: What the report details string would have been, e.g., "1 agency domain(s) found in violation: example.com"

For tests that are not omitted, those keys will be set to "N/A". See screenshot below for an example.

💭 Motivation and context

Closes #1458.

🧪 Testing

  • Add 12 new unit tests
  • Ran ScubaGear with a control omitted, ensured that
    • The HTML output is unaffected by the new JSON keys
    • The OmittedEvaluationResult and OmittedEvaluationDetails values are correct for the omitted control.
    • The OmittedEvaluationResult and OmittedEvaluationDetails values are "N/A" for the controls that were not omitted.
  • Compared the results with reports produced on main for two test tenants, verified that the results are consistent.

📷 Screenshots (if appropriate)

New keys indicated in red:
image

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • PR targets the correct parent branch (e.g., main or release-name) for merge.
  • Changes are limited to a single goal - eschew scope creep!
  • Changes are sized such that they do not touch excessive number of files.
  • All future TODOs are captured in issues, which are referenced in code comments.
  • These code changes follow the ScubaGear content style guide.
  • Related issues these changes resolve are linked preferably via closing keywords.
  • All relevant type-of-change labels added.
  • All relevant project fields are set.
  • All relevant repo and/or project documentation updated to reflect these changes.
  • Unit tests added/updated to cover PowerShell and Rego changes.
  • Functional tests added/updated to cover PowerShell and Rego changes.
  • All relevant functional tests passed.
  • All automated checks (e.g., linting, static analysis, unit/smoke tests) passed.

✅ Pre-merge checklist

  • PR passed smoke test check.

  • Feature branch has been rebased against changes from parent branch, as needed

    Use Rebase branch button below or use this reference to rebase from the command line.

  • Resolved all merge conflicts on branch

  • Notified merge coordinator that PR is ready for merge via comment mention

  • Demonstrate changes to the team for questions and comments.
    (Note: Only required for issues of size Medium or larger)

✅ Post-merge checklist

  • Feature branch deleted after merge to clean up repository.
  • Verified that all checks pass on parent branch (e.g., main or release-name) after merge.

@adhilto adhilto added the enhancement This issue or pull request will add new or improve existing functionality label Feb 28, 2025
@adhilto adhilto added this to the Marlin milestone Feb 28, 2025
@adhilto adhilto requested review from ehaines1 and buidav February 28, 2025 19:38
@adhilto adhilto self-assigned this Feb 28, 2025
@adhilto adhilto linked an issue Feb 28, 2025 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement This issue or pull request will add new or improve existing functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Omitted Rego Evaulation results to the ScubaResults.json
1 participant